Also, rename 'checked' -> 'Checked'

* Create a SuperPMI collection pipeline

Start with a clone of jitstress.yml, and go from there.

* Only build Windows_NT_x64, disable test builds and runs

* TEMPORARY: disable normal PR triggered jobs

* some template files for superpmi

* Little trial error# 1

* fix the superpmi-setup.ps1

* Disable CoreClrTestBuildHost, fix the project path in superpmi-send-to-helix

* Skip build, download, create core_root

* Just copy superpmi scripts, import pipeline-logging-functions

* add a dummy step

* add missing space before message

* Replace message with script

* Temporarily disable dependsOn

* disable another dependsOn

* Pass superpmi.proj file

* logging around send-to-helix

* fix superpmi.proj to include ItemGroup

* have placeholder.txt in WorkItem folder

* fix workitem folder creation, script name

* Re-enable build coreclr, libraries, download, core_root

* fix superpmi.py path

* clone jitutils, build and copy pmi.dll

* Pass Core_Root as payload

* Use hardcoded checked

* Use .dotnet from runtime repo
- Build jitutils before and send the payload to helix

* Fix the pmi.dll location and add  -output_mch_path

* fix the output path

* Try publishing SourcesDirectory

* Upload to HELIX_WORKITEM_UPLOAD_ROOT

* Include test in collection and partition core_root and test

* Enable CoreClrTestBuildHost

* Add dependsOn P1 test

* Add DownloadFilesFromResults

* Properly copy the Tests

* Workaround for dotnet/arcade#6179

* Disable test run, SuperPMI just system.web.dll

* Fix the mch path

* Comment partition for test

* Remove commented code

* Increase timeout to 12 hours

* fast pass

- Comment build coreclr/libraries job
- Instead just download coreclr from some previous run, and tell that it is core_root

* Update the machine pool

* Add a dummy step

* Use SuperPMI_Min.zip that contains all the required files

* Another way of Extracting cab

* download in powershell, extract files separately

* fix the zip path

* Another logic of download zip

* fix path

* fix newdir path

* more fixes

* fix extract step

* add test python script and modify creator

* Map secret to env

* decrease timeout, fix superpmi path in proj, print unzipped contents

* Add a step to download and dir artifacts

* Attempt to call super_pmi upload

* list all contents

* Also dir SourcesDir\artifacts

* Use DownloadPipelineArtifact

* Use the right key

* Upload from helixresults

* cleanup run-superpmi-job.yml

* Re-enable superpmi for CORE_ROOT

* fix the dependsOn

* Replace checked with $(BuildConfig)

* Replace checked with $(BuildConfig)
Comment download artifacts

* change helix queue for windows and add for arm64

* Use FileSeparateChar in superpmi.proj

* Add Linux x64

* Fix the helix queue, property erroro

* logging in shell

* Fix PayloadDir and CorrelationPayloadDir

* Make timeout to 5 hours

* Print workitem contents

* fix WorkItemDirectory

* Just copy pmi.dll

* fix the pmi_assemblies path

* disable reporter

* pop put JitUtilsDirectory before deleting it

* chmod for superpmi-setup.sh

* just copy pmi.dll for unix

* fix the parameter to unix

* Add logging to understand -pmi_assemblies issue

* Add "binaries" folder where payload is present
fix unit script that initializes array

* remove precommand

* Pass Core_Root as CorrelationPayload

- Pass Core_Root as CorrelationPayload
- Pass tag to name .mch file with
- Changes to .ps1 and .sh to copy Core_Root directly

* Add back the workaround for dotnet/arcade#6179

Also, rename 'checked' -> 'Checked'

* fix checked for Linux

* mkdir for linux

* fix path in HelixResultsDestinationDir

* Enable Windows x86/arm64

* fix the python path

* First version of superpmi-setup.py

* Attempt to use superpmi-setup.py

* Fix spelling of LibrariesArtifacts, set python path

* TestArtifacts commented

* superpmi-setup script for linux

* Enable Linux_x64

* Added pydocs

* Add option for exclude_directories

* Moved all the logic to superpmi-setup.py

* Replace superpmi-setup.ps1 and superpmi-setup.sh with superpmi-setup.py

* Fix up in superpmi-setup.py

* fix display name

* add missing spacE

* Use differnet conditional

* ignore PermissionError while cleaning up temp folder

* fix workitem copy for linux

* fix copy for unix

* Publish logs

* Print LibrariesArtifacts

* remove space in front of publish logs desc

* Add missing quotes for LibrariesArtifacts

* Download x64 coreclr to make upload work

* Enable Windows x86/arm64, Linux arm/arm64

* Delete superpmi-setup.ps1, superpmi-setup.sh

* Fix space around LibrariesArtifacts

* Download real coreclr x64

* ignore native dlls

* Make superpmi.py Python3.6 friendly

* Minor cleanup

* Cleanup superpmi-send-to-helix to use helix-inner-step

* Remove redundant steps

* Merge mch files

* Pass osGroup

* Make superpmi-setup script takes tests_directory

* Enable SPMI collection for P1

* General cleanup

* fix merge_mch pattern for OS

* Log the SPMI collection output and put it in upload_root folder

* Replace robocopy/rsync with shutil.copy2

Note that shutil.copy2 is very slow specially on Linux as compared to rsync.
But I think we are ok to have slowness in return of simpler code.

* fix path of pmi.dll to be copied

* Include linux .so and file_exe files in copy_directories

* Try publishing just .mch file

* Revert "Try publishing just .mch file"

This reverts commit a5312c0.

* Disable tests SPMI collection on 42053

- Tests run times out on Windows
- Some of the partition's collection failed for tests because reply was not clean
- Here are the details when libraries/tests were run:
  https://dev.azure.com/dnceng/internal/_build/results?buildId=832814&view=results
- Hence, disable tests run and just have libraries run for now.
- I didn't remove the test run part but commented the portion of tests collection with
  TODO intentionally so when I have to enable it, I will know what to enable.

* Review feedback

Co-authored-by: Kunal Pathak <[email protected]>